BankChurners Project / Yeoman Yoon¶
In [ ]:
import pandas as pd
import numpy as np
from sklearn import metrics
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
from sklearn.preprocessing import StandardScaler
from sklearn import metrics
#model
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn import tree
from sklearn.tree import DecisionTreeClassifier
#Bagging
from sklearn.ensemble import BaggingClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
import warnings
warnings.filterwarnings("ignore")
#Boosting
from sklearn.ensemble import AdaBoostClassifier, GradientBoostingClassifier
from xgboost import XGBClassifier
from sklearn.ensemble import StackingClassifier
#Tuning
from time import time
from scipy.stats import randint as sp_randint
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import RandomizedSearchCV
from sklearn.datasets import load_digits
from sklearn.pipeline import Pipeline, make_pipeline
Import Data¶
In [ ]:
bank = pd.read_csv("BankChurners.csv") # use copy to store the original data
data = bank.copy()
In [ ]:
data.head()
In [ ]:
print(f"The data has {data.shape[0]} rows and {data.shape[1]} columns")
In [5]:
data.info()
Data does not have any missing values
Data Dictionary¶
Data Dictionary:
- CLIENTNUM: Client number. Unique identifier for the customer holding the account
- Attrition_Flag: Internal event (customer activity) variable - if the account is closed then "Attrited Customer" else "Existing Customer" (Target value)
- Customer_Age: Age in Years
- Gender: Gender of the account holder
- Dependent_count: Number of dependents
- Education_Level: Educational Qualification of the account holder - Graduate, High School, Unknown, Uneducated, College(refers to a college student), Post-Graduate, Doctorate.
- Marital_Status: Marital Status of the account holder
- Income_Category: Annual Income Category of the account holder
- Card_Category: Type of Card
- Months_on_book: Period of relationship with the bank
- Total_Relationship_Count: Total no. of products held by the customer
- Months_Inactive_12_mon: No. of months inactive in the last 12 months
- Contacts_Count_12_mon: No. of Contacts between the customer and bank in the last 12 months
- Credit_Limit: Credit Limit on the Credit Card
- Total_Revolving_Bal: The balance that carries over from one month to the next is the revolving balance
- Avg_Open_To_Buy: Open to Buy refers to the amount left on the credit card to use (Average of last 12 months)
- Total_Trans_Amt: Total Transaction Amount (Last 12 months)
- Total_Trans_Ct: Total Transaction Count (Last 12 months)
- Total_Ct_Chng_Q4_Q1: Ratio of the total transaction count in 4th quarter and the total transaction count in 1st quarter
- Total_Amt_Chng_Q4_Q1: Ratio of the total transaction amount in 4th quarter and the total transaction amount in 1st quarter
- Avg_Utilization_Ratio: Represents how much of the available credit the customer spent
In [6]:
data.drop(["CLIENTNUM"], axis=1, inplace = True) # Drop customer unique info, we are not using this
In [7]:
data.describe().T
Out[7]:
In [8]:
data.nunique().sort_values(ascending=False)
Out[8]:
Datatype Conversion¶
For the memory purpose, Convert object to category.
In [9]:
to_category = ['Education_Level', 'Marital_Status', 'Income_Category', 'Card_Category', 'Gender',
'Total_Relationship_Count', 'Months_Inactive_12_mon', 'Contacts_Count_12_mon', 'Dependent_count']
for col in to_category:
data[col]=data[col].astype('category')
Since the target value (Attrition_Flag) has only 2 categories, It is reasonable to convert them to 0 and 1
In [10]:
data['Attrition_Flag'] = data['Attrition_Flag'].replace('Attrited Customer',0)
data['Attrition_Flag'] = data['Attrition_Flag'].replace('Existing Customer',1)
In [11]:
data.info()
In [12]:
all_col = ['Attrition_Flag','Customer_Age','Gender','Dependent_count','Education_Level','Marital_Status',
'Income_Category','Card_Category','Months_on_book','Total_Relationship_Count','Months_Inactive_12_mon',
'Contacts_Count_12_mon','Credit_Limit','Total_Revolving_Bal','Avg_Open_To_Buy','Total_Amt_Chng_Q4_Q1',
'Total_Trans_Amt','Total_Trans_Ct','Total_Ct_Chng_Q4_Q1','Avg_Utilization_Ratio']
for col in all_col:
print(f"{col} = {data[col].unique()}")
print()
In [13]:
small_num_col = ['Customer_Age','Credit_Limit','Total_Revolving_Bal','Avg_Open_To_Buy','Total_Amt_Chng_Q4_Q1',
'Total_Trans_Amt','Total_Trans_Ct','Total_Ct_Chng_Q4_Q1','Avg_Utilization_Ratio']
big_num_col = ['Dependent_count','Months_on_book','Total_Relationship_Count','Months_Inactive_12_mon',
'Contacts_Count_12_mon']
cat_col = ['Education_Level', 'Marital_Status', 'Income_Category', 'Card_Category', 'Gender', 'Attrition_Flag']
In [14]:
for i in small_num_col:
sns.distplot(data[i])
plt.show()
In [15]:
for i in big_num_col:
sns.countplot(data[i])
plt.show()
In [16]:
for i in cat_col:
sns.countplot(data[i])
plt.show()
Missing Value Treatment ¶
Although there is no Nan values, unknown values are considered to be missing values. However, since they are still accounted and recorded in data collecting process, unknown data may represent as another category (may reflect customers' personalities etc.). Hence, no missing treatment handling is done.
Error and Outlier Treatment ¶
After soft EDA, couple of errors/outliers were detected.
- For Credit Limit, $34,516 is the maximum credit limit.
- Months_on_book has too many data that is 36
- Total_Amt_Chng_Q4_Q1 and Total_Ct_Chng_Q4_Q1 have outliers (>3) (DROP)
In [17]:
data[data['Credit_Limit']>33000]
Out[17]:
In [18]:
data[data['Months_on_book']==36]
Out[18]:
In [19]:
data[data['Total_Amt_Chng_Q4_Q1']>3]
Out[19]:
In [20]:
data[data['Total_Ct_Chng_Q4_Q1']>3]
Out[20]:
In [21]:
data=data.drop([1,8,12,269,773])
Univariate Conclusion:¶
- The lowest age of the customer is 26.
- Months_on_book has too many data that is 36. Either some big promotion happened 3 years or some measurement stopped recording after 3 years.
- We have more existing customer. Consider the data balance
- Serious imbalance in Card_Category
In [22]:
data.corr()
Out[22]:
In [23]:
plt.subplots(figsize=(10,10))
sns.heatmap(data.corr(), annot =True, linewidth=1)
Out[23]:
In [24]:
sns.pairplot(data)
Out[24]:
In [25]:
for i in small_num_col:
plt.figure(figsize=(10,5))
sns.violinplot(x=data['Attrition_Flag'], y=data[i])
plt.show()
In [26]:
for i in cat_col:
sns.countplot(data['Attrition_Flag'], hue = data[i])
plt.show()
Bivariate Conclusion:¶
- revolving balance below 500 leads to more cancellation of credit card.
- less than 60 frequent use of credit card leads to more cancellation of credit card.
- lower than 4k of total transaction leads to more cancellation of credit card.
- gender, income, marital status, and education level doesn't seem to effect cancelling credit cards more.
Data Split¶
In [27]:
X = data.drop(['Attrition_Flag'],axis=1)
X = pd.get_dummies(X,drop_first=True)
y = data['Attrition_Flag']
In [28]:
x_train, x_test, y_train, y_test =train_test_split(X, y, test_size=0.3, random_state=1,stratify=y)
print(x_train.shape, x_test.shape)
In [29]:
print(f"Class distritution: {y.value_counts(1)}")
print("")
print("Test Class distritution: {y_test.value_counts(1)}") # good distribution
Function Implementaion¶
In [30]:
def get_score_acc_rec_prec(model):
'''
model : Classifier to predict
[0:train_acc,1:test_acc,2:train_recall,3:test_recall,4:train_precision,5test_precision]
'''
list = [] # defining an empty list to store all scores
pred_train = model.predict(x_train)
pred_test = model.predict(x_test)
#Create Scores
train_acc = model.score(x_train,y_train)
test_acc = model.score(x_test,y_test)
train_recall = metrics.recall_score(y_train,pred_train)
test_recall = metrics.recall_score(y_test,pred_test)
train_precision = metrics.precision_score(y_train,pred_train)
test_precision = metrics.precision_score(y_test,pred_test)
# Add accuracy in the list
list.append(train_acc)
list.append(test_acc)
list.append(train_recall)
list.append(test_recall)
list.append(train_precision)
list.append(test_precision)
return list # returning the list with train and test scores
In [31]:
def print_score(list,what="all"):
''''''
if what == "acc":
print("Accuracy on training set : ", list[0])
print("Accuracy on test set : ", list[1])
elif what == "rec":
print("Recall on training set : ", list[2])
print("Recall on test set : ", list[3])
elif what == "prec":
print("Precision on training set : ", list[4])
print("Precision on test set : ", list[5])
else:
print("Accuracy on training set : ", list[0])
print("Accuracy on test set : ", list[1])
print("Recall on training set : ", list[2])
print("Recall on test set : ", list[3])
print("Precision on training set : ", list[4])
print("Precision on test set : ", list[5])
In [32]:
def make_cm(model,y_actual,labels=[1, 0]):
'''
model : classifier to predict
y_actual : ground truth (y_test)
'''
y_predict = model.predict(x_test)
cm=metrics.confusion_matrix( y_actual, y_predict, labels=[1, 0])
df_cm = pd.DataFrame(cm, index = [i for i in ["1","0"]],
columns = [i for i in ["Predict 1","Predict 0"]])
# annotation=[cm]
# [['1116\n76.18%' '73\n4.98%']
# ['91\n6.21%' '185\n12.63%']]
sns.heatmap(df_cm, annot=True, fmt='')
Model Building¶
Logistic Regression¶
In [33]:
lg = LogisticRegression(random_state=1)
lg.fit(x_train,y_train)
Out[33]:
In [34]:
make_cm(lg,y_test)
In [35]:
lg_score = get_score_acc_rec_prec(lg)
print_score(lg_score,'all')
In [36]:
dTree = DecisionTreeClassifier(criterion='gini',random_state=1)
dTree.fit(x_train, y_train)
Out[36]:
In [37]:
make_cm(dTree,y_test)
In [38]:
dTree_score = get_score_acc_rec_prec(dTree)
print_score(dTree_score,'all')
Random Forest¶
In [39]:
rForest = RandomForestClassifier(random_state=1)
rForest.fit(x_train,y_train)
Out[39]:
In [40]:
make_cm(rForest,y_test)
In [41]:
rForest_score = get_score_acc_rec_prec(rForest)
print_score(rForest_score,'all')
Bagging Classifier¶
In [42]:
bClassifier = BaggingClassifier(random_state=1)
bClassifier.fit(x_train,y_train)
Out[42]:
In [43]:
make_cm(bClassifier,y_test)
In [44]:
bClassifier_score = get_score_acc_rec_prec(bClassifier)
print_score(bClassifier_score,'all')
In [45]:
adaBoost = AdaBoostClassifier(random_state=1)
adaBoost.fit(x_train,y_train)
Out[45]:
In [46]:
make_cm(adaBoost,y_test)
In [47]:
adaBoost_score = get_score_acc_rec_prec(adaBoost)
print_score(adaBoost_score,'all')
Gradient Boosting Classifier¶
In [48]:
gBoost = GradientBoostingClassifier(random_state=1)
gBoost.fit(x_train,y_train)
Out[48]:
In [49]:
make_cm(gBoost,y_test)
In [50]:
gBoost_score = get_score_acc_rec_prec(gBoost)
print_score(gBoost_score,'all')
XGBoost Classifier¶
In [51]:
xgBoost = XGBClassifier(random_state=1, eval_metric='logloss')
xgBoost.fit(x_train,y_train)
Out[51]:
In [52]:
make_cm(xgBoost,y_test)
In [53]:
xgBoost_score = get_score_acc_rec_prec(xgBoost)
print_score(xgBoost_score,'all')
Compare Models¶
In [54]:
#list of all the models
modelList = [lg,dTree,rForest,bClassifier,adaBoost,gBoost,xgBoost]
acc_train = []
acc_test = []
rec_train = []
rec_test = []
prec_train = []
prec_test = []
for model in modelList:
score = get_score_acc_rec_prec(model)
acc_train.append(score[0])
acc_test.append(score[1])
rec_train.append(score[2])
rec_test.append(score[3])
prec_train.append(score[4])
prec_test.append(score[5])
In [55]:
comparison_frame = pd.DataFrame({'Model':['Logistic Regression','Decision Tree','Random Forest','Bagging Classifier',
'AdaBoost Classifier','Gradient Boosting Classifier','XGBoost Classifier'],
'TrainAccuracy': acc_train,'TestAccuracy': acc_test,
'TrainRecall': rec_train,'TestRecall': rec_test,
'TrainPrecision': prec_train,'TestPrecision': prec_test})
comparison_frame
Out[55]:
Choose Random Forest, AdaBosting, xg Boosting.
(Precision is more important in this case)
Reasons:
Want to see the Hyperparameter tuning for Random Forest and see if it can drastically improve.
xgboosting and adaboosting has higher performance without clear overfitting.
Random Forest HyperParameter Tuning¶
GridSearch¶
In [56]:
param_grid_rForest = {"max_depth": [2, None],
"max_features": [1, 3, 10],
"min_samples_split": [2, 3, 8],
"min_samples_leaf": [1, 3, 7],
"bootstrap": [True, False],
"criterion": ["gini", "entropy"]}
In [57]:
%%time
grid_search_rForest = GridSearchCV(rForest, param_grid=param_grid_rForest)
grid_search_rForest.fit(X, y)
Out[57]:
In [65]:
print(f"Best Parameters:{grid_search_rForest.best_params_} \nScore: {grid_search_rForest.best_score_}")
In [69]:
#grid_search_rForest.cv_results_['mean_test_score']
#grid_search_rForest.best_estimator_
RandomSearch¶
In [70]:
param_dist_rForest = {"max_depth": [2, None],
"max_features": sp_randint(1, 12),
"min_samples_split": sp_randint(2, 12),
"min_samples_leaf": sp_randint(1, 12),
"bootstrap": [True, False],
"criterion": ["gini", "entropy"]}
In [71]:
%%time
random_search_rForest= RandomizedSearchCV(rForest, param_distributions=param_dist_rForest, n_iter=10) #default cv = 3
random_search_rForest.fit(X, y)
Out[71]:
In [72]:
print(f"Best Parameters:{random_search_rForest.best_params_} \nScore: {random_search_rForest.best_score_}")
AdaBosting HyperParameter Tuning¶
GridSearch¶
In [73]:
%%time
pipeline = make_pipeline(StandardScaler(), AdaBoostClassifier(random_state=1))
param_grid_adaBoost = {
"adaboostclassifier__n_estimators": np.arange(10, 80, 10),
"adaboostclassifier__learning_rate": [0.01,0.05,0.1,0.3,1],
"adaboostclassifier__base_estimator": [
DecisionTreeClassifier(max_depth=1, random_state=1),
DecisionTreeClassifier(max_depth=2, random_state=1),
DecisionTreeClassifier(max_depth=3, random_state=1),
],
}
# Use Accuracy since both recall and precision are equally important in reality.
scorer_adaBoost = metrics.make_scorer(metrics.accuracy_score)
grid_search_adaBoost = GridSearchCV(estimator=pipeline, param_grid=param_grid_adaBoost,
scoring=scorer_adaBoost, cv=5, n_jobs = -1)
grid_search_adaBoost.fit(x_train, y_train)
print(f"Best Parameters:{grid_search_adaBoost.best_params_} \nScore: {grid_search_adaBoost.best_score_}")
RandomSearch¶
In [74]:
%%time
pipeline1 = make_pipeline(StandardScaler(), AdaBoostClassifier(random_state=1))
param_dist_adaBoost = {
"adaboostclassifier__n_estimators": np.arange(10, 80, 10),
"adaboostclassifier__learning_rate": [0.01,0.05,0.1,0.3,1],
"adaboostclassifier__base_estimator": [
DecisionTreeClassifier(max_depth=1, random_state=1),
DecisionTreeClassifier(max_depth=2, random_state=1),
DecisionTreeClassifier(max_depth=3, random_state=1),
],
}
# Use Accuracy since both recall and precision are equally important in reality.
scorer_adaBoost1 = metrics.make_scorer(metrics.accuracy_score)
random_search_adaBoost = RandomizedSearchCV(estimator=pipeline1, param_distributions=param_dist_adaBoost, n_iter=50,
scoring=scorer_adaBoost1, cv=5, random_state=1)
random_search_adaBoost.fit(x_train,y_train)
print(f"Best parameters are {random_search_adaBoost.best_params_} \nscore={random_search_adaBoost.best_score_}:")
XGBoost HyperParameter Tuning¶
GridSearch¶
In [ ]:
%%time
pipeline2=make_pipeline(StandardScaler(), XGBClassifier(random_state=1,eval_metric='logloss'))
param_grid_XGBoost={
'xgbclassifier__n_estimators':np.arange(50,250,50),'xgbclassifier__scale_pos_weight':[0,1,3,7],
'xgbclassifier__learning_rate':[0.01,0.05,0.1,0.2], 'xgbclassifier__gamma':[0,2,5],
'xgbclassifier__subsample':[0.7,0.85,1]
}
scorer_XGBoost = metrics.make_scorer(metrics.accuracy_score)
grid_search_XGBoost = GridSearchCV(estimator=pipeline2, param_grid=param_grid_XGBoost,
scoring=scorer_XGBoost, cv=5, n_jobs = -1)
grid_search_XGBoost.fit(x_train,y_train)
print(f"Best parameters are {grid_search_XGBoost.best_params_} \nscore={grid_search_XGBoost.best_score_}")
RandomSearch¶
In [ ]:
%%time
pipeline3=make_pipeline(StandardScaler(),XGBClassifier(random_state=1,eval_metric='logloss', n_estimators = 50))
#Parameter grid to pass in RandomizedSearchCV
param_dist_XGBoost={
'xgbclassifier__n_estimators':np.arange(50,250,50),
'xgbclassifier__scale_pos_weight':[0,1,3,7],
'xgbclassifier__learning_rate':[0.01,0.05,0.1,0.2],
'xgbclassifier__gamma':[0,2,5],
'xgbclassifier__subsample':[0.7,0.85,1],
'xgbclassifier__max_depth':np.arange(1,8,1),
'xgbclassifier__reg_lambda':[0,1,3,7]
}
scorer_XGBoost1 = metrics.make_scorer(metrics.accuracy_score)
random_search_XGBoost = RandomizedSearchCV(estimator=pipeline3, param_distributions=param_dist_XGBoost, n_iter=50,
scoring=scorer_XGBoost1, cv=5, random_state=1)
random_search_XGBoost.fit(x_train,y_train)
print(f"Best parameters are {random_search_XGBoost.best_params_} with CV score={random_search_XGBoost.best_score_}:")
Comparing the performance: ¶
It seems like the XGBoost has the best accuracy. However, it is very important to know, with big data, the time complexity can be very costly. Recommend using the second best model to update more often. (24min XGBoost with <1% higher accuracy vs 2 min adaboost with good enough accuracy)
Conclusion & Insights:
Since the purpose of the project is to find the customers who are likely to stop using credit card, it is also important to check the customers' behaviors. From EDA, it became more clear that:
- revolving balance below 500 leads to more cancellation of credit card.
- less than 60 frequent use of credit card leads to more cancellation of credit card.
- lower than 4k of total transaction leads to more cancellation of credit card.
- gender, income, marital status, and education level doesn't actively effect cancelling credit cards.
Consider the following business recommendations:
- call the customer status when their revolving balance is too low or going negative. and see how bank can help
- run more promotions to increase the natural credit card usage
- run a survey from customers and also notify them cancelling credit card temporarily negatively affects credit score.
In [ ]: